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ABSTRACT 

An ACER (Australian Council for Educational Research) study 
on literacy and numeracy was established in 1999 with 1000 children from a 
random national sample of 100 schools. The children in the study began their 
schooling at the beginning of 1999. Many students transferred to other 
schools since the study began and have been tracked in their new schools, and 
so there is a larger number of Australian schools in the project. The study's 
main purpose is to monitor the growth of these children in literacy and 
numeracy throughout their primary schooling. A major outcome of the study 
will be empirically derived scales showing typical progression of development 
in literacy and numeracy. They will constitute progress maps for literacy and 
numeracy. In the study's first three years, covering the "early years" of 

schooling, students were assessed by their teachers with sets of literacy and 

numeracy tasks designed at ACER. Assessments were conducted in one-to-one 
interviews, with the teacher recording students' responses on a marking 
guide. So far, five sets of linked assessment tasks have now been designed to 

cover the expected range of what students know and can do in literacy and 

numeracy in the early years of schooling. This paper focuses on the processes 
involved in developing these tasks. The longitudinal study reported in the 
paper is underpinned by the concept of developmental assessment- 
developmental assessment supports teachers' use of assessment data as a basis 
for teaching, designing curriculum, and planning classroom organization to 
meet individual needs. (NKA) 
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BETTER THAN BEIGE: DESIGNING ASESSMENT TASKS TO ENHANCE 
LEARNING AND MEASURE GROWTH IN THE EARLY YEARS OF SCHOOL 



Paper presented to the Australian Association for Research in Education conference, 

Fremantle,WA, 2-6 December, 2001 

7 think of our school as beige. We tend to be the average of 'like' schools. Can we do better 
than that?’ Assessment tasks that demonstrate achievement and growth in student 
learning can assist schools to improve teaching practices and enhance students' learning 

opportunities. 

THE ACER LONGITUDINAL LITERACY AND NUMERACY STUDY 

This ACER study was established in 1999, with 1,000 children from a random national 
sample of 100 schools. The children in the study commenced school at the beginning of 
1999. Since the commencement of the study, many students have transferred to other 
schools, for a variety of reasons. We have tracked many of these students to their new 
schools, and so now have a much larger number of Australian schools in the project. 

The main purpose of the study is to monitor the growth of these children in literacy and 
numeracy throughout their primary schooling. A major outcome of the study will be 
empirically derived scales showing typical progression of development in literacy and 
numeracy. These scales will provide a framework for reporting repeated measures of 
achievement, thus showing growth over time. They will constitute progress maps for 
literacy and numeracy. 

In the first three years of the study, covering the 'early years' of schooling, students 
were assessed by their teachers with sets of literacy and numeracy tasks designed at 
ACER. The assessments were conducted in one-to-one interviews, with the teacher 
recording students' responses on a marking guide. In the first two years of school, these 
assessments were carried out close to the beginning of the school year, and towards the 
end of the year. In the third year, the students were assessed once, in Term 2. Thus five 
sets of linked assessment tasks have now been designed to cover the expected range of 
what students know and can do in literacy and numeracy in the early years of 
schooling. This paper focuses on the processes involved in developing these tasks. 
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DEVELOPMENTAL ASSESSMENT 



This longitudinal study is underpinned by the concept of developmental assessment. 
Developmental assessment supports teachers' use of assessment data as a basis for 
teaching, designing curriculum, and planning classroom organisation to meet 
individual needs. The unique feature of developmental assessment is its use of a progress map 
or continuum. A progress map describes the nature of development - or progress or growth - in 
an area of learning and so provides a frame of reference for monitoring individual development. 
(Masters and Forster, 1996, p 1). A progress map outlines skills, understandings and 
knowledge, in the sequence in which they typically develop. Assessment tasks are 
selected to address the range of difficulty and the scope of the desired progression. The 
process of obtaining evidence, through assessment, to support inferences about 
students' achievement in relation to a progress map is known as developmental 
assessment. 

Empirically derived scales or progress maps are constructed through analysis of 
assessment data designed to address a range of achievement in the development of 
related skills. Analysis of the data using Item Response Theory (Rasch model) 
determines the relative difficulty of the skills addressed through the assessment. The 
evidence provided from the assessment data about students' skills and knowledge is 
calibrated onto a scale that illustrates a typical progression of development. The vertical 
scale begins with the skill students are most likely to demonstrate and ends with the 
skill students are least likely to demonstrate. The location of skills on the scale indicates 
the relative difficulty of those scales. Skills that are clustered together are of similar 
difficulty. The further apart the skills, the greater the difference in their relative 
difficulty. 

Describing the progression of development represented by the skills calibrated onto the 
scale requires a broader interpretation than itemising the surface content of the 
assessment data used to construct the scale. Evidence for the skills calibrated onto the 
scale may come from the different items or questions used in the assessment or the 
levels of a scoring rubric used, for example, to assess writing or an oral presentation. To 
be a useful description of development on the scale, skills must be described in a way 
that both generalises the original question or task and also identifies particulars about 
the question or task that make it relatively easy or difficult compared to similar tasks. 
Assessment tasks are inevitably set within a context. For example, a question may be 
asked about the relationship between particular characters in a text, but the purpose of 
the question is to assess an underlying ability to comprehend the text. However, some 
aspects of the context of the task are important. If the relationship between the 
characters is directly stated in the text this would make the task easier than if the 
relationship is implied. If the text was a poem rather than a narrative this may make the 
task more difficult. Descriptions of each skill on the scale need to identify the key 
features that contribute to its location in the sequence of development. These 
descriptions of skills are aligned alongside the scale to illustrate a typical progression of 
development. 
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Once a scale has been constructed, teachers can use the same assessment tasks to locate 
student achievement on the scale. Teachers can administer the assessment, follow the 
scoring instructions, calculate a student's overall score and locate that student's 
performance on the scale. Related skills the student is likely to demonstrate are 
described below the location of the student's performance on the scale and the skills the 
student is less likely to demonstrate are described above the location on the scale. The 
scale helps the teacher to locate the next teaching steps for each student, including those 
who may have difficulty and require early intervention, and those who require 
extension beyond the skills demonstrated by the average student in the class. 

The LLANS provides an example of a set of linked developmental assessments 
designed to be calibrated onto a common scale for literacy and a common scale for 
numeracy. At this point in the study, a scale for literacy has been constructed based on 
the first four sets of literacy tasks, describing a typical progression of development in 
literacy for the first two years of schooling. From the first three sets of numeracy tasks, a 
scale for numeracy has been constructed that describes a typical progression of 
development in numeracy skills from the beginning of the first year to the beginning of 
the second year of school. 

Figure 1 shows the current development for the LLANS literacy scale. (Rowe, K and 
Stephanou, A, 2001). Skills are described at the left side of the scale. These descriptions 
summarise clusters of related skills of similar levels of difficulty or describe mastery of 
a series of related skills. For example, the description, manipulates beginning, middle and 
end sounds in short words to make new xvords, is located approximately two thirds of the 
way up the scale. Manipulation of beginning sounds only tends to be the easiest skill, 
manipulation of end sounds tends to be more difficult and manipulation middle sounds 
of tends to be the most difficult. However, this pattern is not always consistent. The 
description of manipulating sounds in short words is given on the scale at the point 
where the student is more likely to demonstrate mastery of the manipulation of the 
sounds in short words regardless of the word or where the sound is located. 

On the right of the scale, the 'box and whiskers' graphs show the range of mean 
achievements of the whole cohort of LLANS students. A separate graph is shown for 
each of the four surveys. The shaded section or 'box' represents the range of 
achievement of the middle fifty per cent of students in each survey. The lines or 
'whiskers' above and below the box show the range of achievement of the middle 
eighty per cent of students. The key indicates the icon that would be used to locate a 
student's achievement in each survey - repeated measures which indicate progress over 
time. 
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Longitudinal Literacy and Numeracy Study (LLA.NS) 
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DESIGNING ASSESSMENTS FOR THE EARLY YEARS OF SCHOOL 



The quality of the scales described above is dependent on the quality of the assessment 
tasks from which they derive. The centrality of the development of scales in the LLANS 
study created an imperative to develop high quality tasks. The development of the 
assessment tasks for early learners in the first three years of schooling has constituted 
an important phase of the project, which, at the end of 2001, is now completed. From 
2002, when the students are in their fourth year of school, the assessments will be pen 
and paper assessments, using the ACER DART (Development Assessment Resource for 
Teachers). 

The tasks were developed through a rigorous process of collaborative work by ACER 
item writers, and trialled in schools. A series of 'panels' of item writers scrutinised draft 
tasks until it was agreed that the tasks were likely to gather a broad range of responses 
from students in relation to particular skills and knowledge targeted by the tasks. The 
trial data indicated where the tasks needed improvements and adjustments, and the 
tasks were revised on the basis of this information. 

Work on the LLANS tasks commenced in 1998, and a number of criteria shaped their 
development. These criteria included: 

• The tasks should be research based, that is, they should assess aspects of literacy 
and numeracy that contemporary research indicates to be central to the 
development of strong literacy and numeracy skills; 

• The tasks should engage students and be built around contexts familiar to early 
years students; 

• The tasks would be administered one to one in an interview situation, if possible 
by the student's own teacher; 

• Where possible, the tasks should involve authentic texts and hands-on 
equipment; 

• The tasks should be easy to for teachers to administer, and supported with clear 
and explicit marking guides; 

• The tasks should be designed to be administered in a reasonable time, taking 
account of the attention span of early years students, and teacher's workload. 

• The tasks should link with the ACER DART tasks available for middle and upper 
primary school. 

Assessments in the early years of school have a number of specific requirements 
because they are administered one to one with young children. It was agreed that the 
LLANS assessments should be child-centred and unobtrusive. The LLANS activities 
were designed to take place in a meaningful context and emphasise process as well as 
product. Particular attention was given to providing administration and scoring 
instructions which were particularly clear to ensure consistency and the reliability of the 
data. 




6 



THE LLANS NUMERACY ASSESSMENT TASKS 



The LLANS numeracy tasks address skills in number, space, measurement and chance 
and data. Each set of LLANS tasks is composed of 5-6 short units, each consisting of 
approximately eight questions of a range of difficulty. Each unit addresses a different 
aspect of numeracy. For example, the fifth set of LLANS numeracy tasks assesses 
different aspects of Space through one unit based around wooden triangles and another 
unit based around a treasure map. Aspects of Measurement are assessed with a unit 
based around wooden blocks. Two other units assess aspects of Number. One unit 
based around bundles of straws assesses place value and a unit based around a 
conventional work sheet assesses patterns and computation skills. 

The LLANS numeracy tasks are based around hands on activities that use every day 
and familiar classroom materials. The tasks are a rich source of diagnostic information 
in the observations the teacher makes of the student manipulating the materials. The 
tasks are structured to focus the child's attention on the pertinent features of the 
materials being used and lead the student from simple to more complex demonstrations 
of understanding. 

The fifth set of numeracy tasks includes a unit which is based around a set of red 
triangles of different sizes that include examples of equilateral, isosceles, and scalene 
triangles. All the questions in this unit focus on the spatial properties of the triangles. 
The triangles are not used as objects for counting. All the triangles are red so that colour 
is not a distracting feature. The order of questions encourages children to consider 
different spatial properties as they handle the triangles. The first question requires the 
student to match each triangle to its outline, the second question establishes if the 
student knows that all the three sided shapes in front of them are all called triangles, the 
third question requires the student to identify which triangle can be rotated and flipped 
in order to fit exactly on top of another triangle. Further questions ask the student to 
sort the triangles into two groups describing the criteria for the sort and to fit two given 
triangles (right angled) together to make a rectangle. 

The questions selected for each survey have a wide range of difficulties to provide 
evidence to describe growth. The difficulty of the questions is considered in ordering 
the questions in the survey booklet. Each unit begins with one or two easy questions to 
build students' confidence. After that, more difficult questions are interspersed with 
easier questions, to encourage students to attempt every task. 

The administration and scoring instructions used in the LLANS assessment tasks are 
simple, clear and straightforward. The style of administration is consistent for all tasks. 
All the information the teacher requires to administer and score each question is on one 
page. The LLANS tasks are sequenced to use any equipment in a logical order. Where 
possible, setting up requirements are incorporated into a simple initial question which 
encourages the student to handle the materials confidently. For example, in the 
triangles unit the first question asks the child to match each triangle to its outline. This 
is a simple task that encourages the child to pick up each triangle and also ensures that 
the orientation of the triangles is standardised for the two subsequent questions. If the 
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child is unable to match the triangles correctly, the teacher models the process before 
continuing to the next question. 

Figure 2 shows an example of a question from the fifth set of LLANS numeracy tasks. 



Ruler 

(inside 

envelope) 



Put ruler and triangle A in 
front of child. 

Run your finger right along 
the longest side from left to 
right and say 

How long is this? 




Prompt the child for the units if 
they don't give them. Say 
" fifteen (or other number 
what?" 





















fifteen cm 


fifteen cm when prompted 
for units 


fifteen (doesn't know units 
when 


prompted) 


incorrect value with cm 


incorrect value without 
units 



no attempt 



Figure 2: Question from fifth set of LLANS numeracy tasks 



This example illustrates standard features of LLANS tasks. The marking guide for each 
question is located next to the question. The marking guide addresses most possible 
student responses ensuring that the teacher is guided in how to score different 



responses. The levels of the marking guide also support teachers' diagnostic 
interpretations of students' performances. 

Figure 2 shows the layout of the survey booklet with the columns of boxes used to score 
students' responses (note that the text has been reduced for this example). Up to five 
students, numbered student one to student five, can be scored in one survey booklet. 
The score for 'student one', for each question is indicated by a tick in the appropriate 
box in column one, the scores for student two in column two and so on. 



Equipment 




Instructions and 
questions 


Marking guide 




1 




2 




3 




4 




5 






Give the student the 
two D triangles and 
the C triangle. 

Make a rectangle 
using all 

three of these shapes 
























makes a 
rectangle 






















makes a 
different shape 






















other 






















no attempt 























Figure 3: Example of the layout of a LLANS survey booklet 

Instructions to the teacher are shown in italics and instructions to be read aloud to the 
student are shown in bold. All equipment that is required is listed at the beginning of 
the unit and separate pieces listed again next to the question where they are to be used. 

Instructions addressed to the students have the following characteristics: 

• present tense 

• simple sentences 

• command given last in a sequence of instructions 

• brief 

• precise simple vocabulary 

Choice of language is a critical factor in developing items for young students. It is 
imperative that students understand what is being asked of them. Much of the 
development time for the LLANS tasks was devoted to refining instructions that the 
teacher gives to the student to make them as simple and clear as possible. In many 
instances simple everyday language was chosen in preference to mathematical terms 
because the underlying skill that was being assessed was a conceptual understanding or 
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capacity to manipulate objects. For example, in the fifth numeracy survey the student 
has a number of triangles in front of them. The student is told, 'Tivo of these shapes fit 
exactly on top of each other 1 , and then asked 'Which two?' The student was not asked to 
find two congruent shapes as they were not likely to be familiar with this term. 
However, in some instances it was appropriate to assess students' knowledge of 
specialised mathematical language that was more likely to, 'Find a way to sort the 
shapes into two groups.' Previous survey results had suggested that many students in 
their first year of school were not able to demonstrate sorting objects even when sorting 
objects had been modelled prior to the question. Anecdotal evidence suggests that 
students may have confused sorting with tidying or making a pattern. This question 
about sorting was given to students at the start of their third year of school, without any 
modelled support, to assess the development of students' understanding of the 
mathematical meaning of sort. 

THE LLANS LITERACY TASKS 

The literacy tasks in LLANS have been developed around phonemic awareness, 
concepts of print, reading fluency, making meaning from text, and writing. They 
include comprehension of the meaning of written and visual texts, knowledge of 
conventions of text, contextual understanding as well as mastery of various aspects of 
the mechanics of text decoding. 

In each set of tasks, a picture story book has provided the context for activities relating 
to the child's understanding of text, and for prompting writing. Looking at and 
listening to picture story books are significant activities within the literacy experiences 
of young children, both in and beyond the classroom. The capacity to understand 
narrative, to interpret the actions of character's, and to understand visual aspects of 
texts are skills which many children develop from an early age. They are multi-modal 
texts, familiar to children. (With the introduction of new information technologies, 
multi-modal texts are increasingly prevalent in everyday literacy.) 

The picture story books were selected to interest a wide range of students. For the 
second set of tasks, administered at the end of the first year of school, the book Handa's 
Surprise by Eileen Browne was chosen. The book features children from the Luo tribe of 
south-west Kenya. A significant component of the meaning of this text is carried by the 
illustrations. The text uses a structure of an accumulation of repeated actions, 
culminating in a surprise in two senses. Handa packs a basket of seven fruits to take to 
her friend Akeyo, in the next village. En route, a monkey, an ostrich, a zebra and other 
animals steal pieces of fruit from her basket, as Handa continues to walk on, unaware of 
what is happening. The written text conveys Handa's thoughts about which fruit her 
friend will enjoy most, while the actions of the various animals are conveyed through 
the illustrations. At the point when the basket is empty, Handa walks under a tangerine 
tree just when a goat buts the tree, and the tangerines tumble in to refill the basket. The 
last lines of text, when Handa meets Akeyo capture the climax: 

"Tangerines!" said Akeyo. "My favourite fruit." 



"TANGERINES?" said Handa. "That is a surprise!" 



The final full page illustration conveys a powerful image of friendship as the girls look 
at each other while they taste the fruit. 

The assessment activities were designed to follow the reading of the book to the student 
by the teacher. The first activity was a retelling: "I want you to tell me what happened 
in the story". The marking guide indicated a range of possible responses: 

• orientates the story; eg, the girl is taking fruit to her friend. 

• Includes complication - taking the fruit by the animals 

• Understands the goat's involvement, eg, fruit falls into her basket. 

• Includes the end of the story, eg, they eat the fruit together. 

• Identifies the surprise at the end. 

The teacher ticked the marking guide for as many of these responses as were evident in 
the retell. 

Subsequent activities focused on details of the text, for example, the title. "The book is 
called Handa's Surprise. Why is Handa surprised?" The double page spread of the goat 
butting the tree provided a useful focus for probing the child's understanding of a 
sequence of illustrations: "What's happening in these pictures?" While looking at these 
pages, the students were also asked "Why is this important in the story?, in order to 
determine the extent to which they could identify the information communicated 
visually. 

The last page, the full page illustration, provided the context for the question, "How are 
the girls feeling?" This was scored on the basis of the students identifying a positive 
emotion, or giving an incorrect response. A second question asked students to identify 
those aspects of the text that conveyed the girls' feelings: "How does the picture show 
you that the two girls are very good friends?" Further questions were related to the 
specific detail in the text, such as the banana skin falling from the tree after the monkey 
has taken the banana from Handa's basket. "What is this? Why is it there?" Finally, 
there was a question about visual conventions, "This picture is a close up. Why has the 
illustrator drawn it in this way?" 

Overall, these activities assess a range of responses to text, within the context of a 
familiar type of text. Having worked through these activities, the students next attempt 
a writing task. The experience of listening to the text read aloud, and of having 
considered aspects of the text in responding to the questions, contextualises the writing, 
and provides some scaffolding for the content of the writing. The prompt for the 
writing was focused on one section of the text - the double page illustration when 
Handa arrives in Akeyo's village. "Look at this picture. Do you remember this part of 
the story? Write about this part of the story. Write about what is happening". The 
scoring rubrics for this activity were clustered into four categories: 

The child reading his/her own writing 
The relevance of the content to the story 
The language features 
Punctuation. 
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The marking guide for the content category provided these options: 

• Relates an event from this part of the story; 

• Relates an event from elsewhere in the story connects to picture only 

• Unrelated to picture or story. 

For language, the following marking options were provided: 

• Writes one or more generally readable sentences 

• Some recognisable words with spaces 

• Contains groups of words and word like spaces 

• A continuous string of letters 

• Scribble 

• No attempt. 

This activity provided the first evidence of writing achievement in the study. Apart 
from asking to students to write their own name, it was decided that it was not 
appropriate to ask students at the very beginning of their first year at school to write. By 
the end of the first year, many students were able to write coherent texts. The following 
samples indicate the range of achievement. Each sample indicates signs of the student's 
growth as a writer. 



fen \d,oh Surprise 

~Tti€r^ Han eta 

a banaCA OivJ, Skr 

■Kke# ran \o 



md 

Wact taricifOLiVD 



This text is constructed from specific, accurate observations of the visual detail in the 
text. It is presented in the form of a correctly punctuated sentence, with three ideas 
linked with 'and'. The spelling of basket ('barscit') is phonetically plausible. 



BEST COPY AVAILABLE 




This second sample suggests that this student has not understood that the fruit was not 
mangoes, but tangerines. However, the sentence is complete, and punctuated with a 
capital letter and a full stop. Words are clearly distinguished, letters are well-formed, 
and the meaning is clear. There is evidence of some self correction, in the 'o' that has 
been crossed through. 




This two sentence response indicates strong understanding of the main point of the 
story. The substitution of 'mandareans' (phonetically plausible spelling) for 'tangerines' 
perhaps indicates that the writer is making links with prior knowledge. There are two 
main ideas in this response, both central to the original text. 
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The fourth sample also presents two ideas. The main action of these two pages is neatly 
summarised - "Handa is giving Akeyo a surprise". This is supported by a comment on 
Akeyo's action of waving to her friend, referring to specific visual detail. 



arvM fwt seven fruits in W - noir 

the m onheu to nek sq. bhww 

01 hmum J u/irt 'nsrri <m. ran atftrj 





01 ftd he eott Jt. 



This response does not relate directly to the specified pages. It goes back to the reason 
for Handa's surprise - the loss of the fruit, beginning with the monkey. The sentence 
construction is strong - beginning with Handa as subject, and then introducing the 
complication of the monkey's action. The detail is specific - 'seven fruits'. Almost all 
words are spelt correctly. We do not know the extent to which the children used the 
picture story book as a resource for spelling. If this were the case, it is indicative of a 
strategic approach to spelling. 
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This response is set up in narrative style: "When Handa cam back which is a most 
appropriate choice of form in response to the task. The piece focuses on the surprise, 
recognising the centrality of this to the whole text. 

Data from these activities has provided the staring point for the writing component of 
the literacy scale. 

CONCLUSION 

The LLANS assessment tasks have provided valid and reliable data for the work of 
developing the described scales. The sequence of psychometric processes involved in 
the development of the scale indicate the 'fit' of the tasks, and the way the data will 
contribute to the scale. The tasks address central aspects of literacy and numeracy, and 
have tapped into the broad range of achievement across the whole cohort of students. 
This spread of achievement is critical to the construction of the scales. 

The tasks have worked in the context of teachers administering and marking their own 
students' work. When the tasks have been sent to schools, very few queries have been 
made regarding their administration by almost a thousand different teachers who have 
assisted in the collection of LLANS data over five sets of tasks, without any special 
training. 

Finally, anecdotal reports from schools indicate that the tasks are providing useful 
models of assessment for early years teachers. 
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